Cooling diversity in data centers

ABSTRACT

A data center cooling system is disclosed. The system includes a plurality of computer racks arranged in a plurality of substantially parallel rows, cooling units associated with the computer racks and arranged in substantially parallel rows to cool air warmed by the cooling racks, and cooling fluid supply and return conduits that are divided by isolation valves into a plurality of cooling sub-loops, wherein adjacent cooling units in a common row are supplied from different sub-loops, and individual sub-loops serve cooling units in multiple rows, so as to provide water-side diversity across the cooling system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority under 35U.S.C. §120 to, U.S. application Ser. No. 12/060,130, filed Mar. 31,2008, which in turn claims priority to U.S. Application Ser. No.60/976,265, filed on Sep. 28, 2007, the contents of both are herebyincorporated by reference in their entirety.

TECHNICAL FIELD

This specification relates to providing utilities such as electricalpower and cooling to a data center.

BACKGROUND

Computer data centers are large commercial buildings that house a verylarge number of computers, such as thousands or tens of thousands ofrack-mounted computer servers and other computing equipment (e.g.,storage devices, power supplies, and networking equipment). Generally,computer data centers provide computing services, such as web pagehosting, search, e-mail, video, and other services, to a number ofdistributed remote users.

Because internet-connected users have grown enormously in numbers, andthe types of services they demand have grown enormously in complexity,data centers must now provide a quickly increasing amount of computingresources. As a result, the size of data centers has increased, thenumber of data centers has increased, and because data centers requirethe use of electronic equipment, the electrical demands of data centershave increased. Because the electricity used by data centers is turnedinto heat, the cooling demands of data centers have increasedsignificantly also.

SUMMARY

This document discusses various mechanisms and techniques forconstructing components of data centers, such as electrical and coolingsub-systems. In certain implementations, the systems and techniques maypermit a shorter time period from the start of construction to the timethat a significant portion of a data center may begin operating. Forexample, leased equipment such as chillers and electrical substations orswitching equipment may be used initially to start operations with aportion of a data center; permanent, owned equipment may be transitionedin as it is installed and as additional computing capacity is added atthe data center. When the permanent equipment is able to handle theload, the leased equipment may be removed and returned.

In another example, air-cooled chillers are installed to provide initialcooling for a portion of a data center before water supply andwastewater permits can be obtained for a facility. The facility can thenoperate at a limited capacity until the appropriate permits are granted.When such permits are granted, cooling towers that have been installed,such as during the delay of the permitting process, may be broughton-line and may begin using water. The air-cooled chillers may then beremoved or may be used to provide supplemental cooling, such as onparticular warm days when cooling tower cooling is insufficient, or whencertain towers are off-line such as for repair. Similar approaches maybe taken using hybrid cooling towers, which may be run in a closed,non-evaporative mode until a sufficient water permit is received, andthen may be run in a higher-capacity, open, evaporative mode after awater permit has been received. Such operation techniques can permit acenter to begin running much more quickly than would otherwise bepossible, particularly when substantial water is needed so that apermitting process would be long and drawn out.

In another example, a cooling system for a data center may be segmented,so that portions of the data center are brought on line, orcommissioned, as they are completed in sequence. Each unit of coolingdevices in the data center, such as one or more rows of cooling devicesassociated with rows of computer racks, may be correlated to aparticular cooling plant, which may include one or more cooling towers,heat exchangers, and associated circulation pumps. The computing devicesmay be installed, the cooling units may be erected and connected, andthe cooling plants may be added at comparable rates to each other, sothat cooling capacity generally expands with computing capacity (andthus with heat generation) during a construction process. Each unit ofadditional computing power and additional cooling capacity, such as arow or rows of computer racks and a single modular cooling plant, may bebrought on-line without substantially interfering with the continuedoperation of already-commissioned portions of the data center.

The systems and techniques described here may provide one or moreadvantages. For example, data centers may be brought on-line much morequickly than would be permitted if an organization needed to wait forcommissioning of multiple portions of a system or all of a data center.Such techniques may operate within constraints that present themselvesin construction-based situations, such as delay times in acquiring andinstalling large-scale utility equipment such as electrical switchingstations, cooling towers, and the like. Rather, by operating withsmaller-scale equipment or limited-capacity equipment during theconstruction process, including equipment that may be leased rather thanowned, problems created by acquisition of and commissioning oflarge-scale equipment can be avoided or at least prevented from directlyaffecting the operation of the data center. Rather, a data center may bebrought on-line in a much more piece-meal fashion that permits thecomputing workload to be started early in the process.

In one implementation, a method of providing utilities to a computerdata center is disclosed. The method includes providing one or morecooling fluid conduits having a plurality of segments that can beisolated from other segments, connecting a first group of computercooling units and a first cooling plant to the conduit and commissioningthe first group of computer cooling units and the first cooling plant,and sequentially commissioning additional groups of computer coolingunits and cooling plants. The additional groups of cooling units may becommissioned without substantially interrupting operation of the firstgroup of computer cooling units. Also, commissioning the additionalgroups of computer cooling units can comprise connecting the additionalgroups of computer cooling units to segments of the one or more coolingfluid conduits.

In one aspect, the cooling plants are modular units each including oneor more cooling towers and associated heat exchangers. Also, each of theone or more fluid conduits can define a substantially constantcross-section along a substantial portion of its length. In addition,the first cooling plant can comprise a rented cooling plant and theadditional cooling plants comprise purchased cooling plants. Also, thesequential commissioning can occur substantially to maintain n groups ofcomputer cooling units generally matched in load to a cooling plant, andn+1 or more cooling plants, so as to provide back-up cooling capacity.

In another implementation, a data center cooling system comprisescooling fluid supply and return conduits having a plurality of segmentsthat can be isolated from other segments of the fluid supply and returnconduits, a first group of computer cooling units inside a data centerand a first cooling plant in operational fluidic connection through thesupply and return conduits, and a second group of computer cooling unitsinside the data center and a second cooling plant in physical connectionwith the supply and return conduits but not in fluidic connection withthe first group of cooling units and the first cooling plant. The supplyand return conduits can be provided with a plurality of taps, whereineach pair of taps corresponds to a group of computer cooling units to beattached to the system. Also, each of the supply and return conduits candefine a substantially constant cross-section along a substantialportion of their lengths.

In some aspects, the first cooling plant comprises a rented coolingplant and the additional cooling plants comprise purchased coolingplants. Moreover, the system can further comprise a plurality of groupsof computer cooling units, wherein each group is connected to a distinctcooling loop, wherein more than one of the plurality of groups ofcomputer cooling units has been commissioned and one or more of theplurality of groups of computer cooling units has not been commissioned.

In yet another implementation, a method of providing utilities to acomputer data center is disclosed. The method comprises leasing one ormore cooling plants for providing utilities to the data center,operating a portion of a data center during construction of the datacenter using the leased cooling plants, and transitioning from theleased cooling plants to owned cooling plants during construction of thedata center. The units may be selected from a group consisting ofelectrical substations, cooling towers, and chillers.

In another implementation, a method of providing utilities to a computerdata center is disclosed. The method can include initially connectingone or more air-cooled chillers to a data center as primary coolingplants, obtaining a government-issued permit, and after obtaining thegovernment-issued permit, transitioning primary cooling for the datacenter to one or more cooling towers. The method can also include, aftertransitioning primary cooling for the data center to one or more coolingtowers, operating the air-cooled chillers to provide supplementalcooling. In addition, pre-water operation may occur using a closedportion of a hybrid cooling tower or other similar system, with the openportion of the cooling tower being used (with evaporative cooling added)after a permit is obtained, so that make-up water may be accessed. Inaddition, the method can include, after transitioning primary coolingfor the data center to one or more cooling towers, disconnecting one ormore of the one or more air-cooled chillers from the data center. Insome aspects, the one or more disconnected air-cooled chillers compriseleased equipment.

The details of one or more implementations of the data centerconstruction systems and techniques are set forth in the accompanyingdrawings and the description below. Other features and advantages of thesystems and techniques will be apparent from the description anddrawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1A is a plan view of a data center during early construction.

FIG. 1B is a plan view of the data center of FIG. 1A at a later stage ofconstruction.

FIG. 1C is a plan view of a data center during early construction thatis using leased cooling plant equipment.

FIG. 2 is a side sectional of a data center.

FIG. 3 is a flowchart showing a process for staged start up of a datacenter facility.

FIG. 4 is a flowchart of a process for transitioning a data center fromtemporary to permanent utility equipment.

FIG. 5 is a flowchart of a process for sequential commissioning of partsof a data center system.

FIG. 6A is a plan view of a data center during construction, showingcomputers with cooling units and a cooling piping system.

FIG. 6B is a plan view of a data center cooling piping system.

FIG. 7A is a plan view of a data center showing continued operation ofcomputers while additional computers are installed.

FIG. 7B is a section view of the data center in FIG. 7A.

FIG. 8 is a conceptual diagram showing the use of portable utilityservices to serve a data center temporarily.

FIGS. 9A and 9B show plan and sectional views, respectively, of amodular data center system.

FIG. 10 shows a schematic diagram of a hybrid cooling tower for use witha data center.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

In general, various example techniques are described here forconstructing and operating data centers in manners that can permitaccelerated operation of the data centers, so that the centers may beginserving customers more quickly than would otherwise be possible. Thetechniques generally involve establishing a cooling plant to handle aportion of a complete data center load, and commissioning that portionof the data center while construction continues on other portions of thedata center. The cooling plant may, in certain examples, include (1)leased equipment that is returned to a leasing company after othercooling plants are completed to supply other portions of the datacenter; (2) lower capacity equipment such as air cooled chillers orhybrid cooling towers that can be leased or can be on-site forsupplemental cooling after a full data center is operational; or (3)modular cooling plants that may be brought online sequentially as theyare completed during a construction process, without substantiallyinterfering with ongoing operation of the data center. In addition,various arrangements discussed here may provide greater flexibility inongoing operation of cooling systems in data centers.

FIG. 1A is a plan view of a data center 100 during early construction.The data center 100 is generally made up of a computing facility 102that may house thousands or tens of thousands of computing systems.Generally, computational demands are made on the data center 100 byremote users distributed throughout a geography, such as by searchrequests, requests to access e-mail applications, or request to accessother applications such as mapping, shopping, and other such services.The data center 100 may be operated, for example, by a company such asGOOGLE or that provides numerous computing services to the public, or bya company that provides more limited services such as e-commerce relatedservices, or alternatively by a company that provides simply access tovarious applications in a server-based model (such as by asoftware-as-a-service distribution model).

The computing facility 102 is served in this example with a utility inthe form of cooling water, by a cooling plant 108 a. The cooling watermay generally be circulated to cooling units located throughout thefacility 102, that include water-to-air heat exchangers for cooling airin the facility 102 that has been heated by computers in the facility at102. Depending on the particular application, cooling fluid such as inthe form of cooling water may be supplied at a first low temperature andreturned to the cooling plant 108 a at a second higher temperature. Thesupplied temperature for the cooling fluid may be in the range, forexample, of 40-70° F. or 40-90° F., and the return cooling fluidtemperature may be in the range of 50-100° F. or 50-130° F., asexamples.

As described in more detail below, the cooling plant 108 a may providecooling to the cooling fluid in a variety of manners. In a relativelyefficient manner, cooling may be provided by cooling towers that removeenergy from a system by water evaporation, so that water returned by thecooling towers is significantly cooler than water supplied to thecooling towers. In general, cooling towers systems are open systems sothat the water in such a system may become contaminated and otherwiseunfit for use in clean piping systems. As a result, cooling plant 108 amay also be provided with a water-to-water heat exchanger or multipleheat exchangers that transfer heat from water returned from facility102, to water from cooling towers.

Various parameters for the data center 100, such as operating airtemperatures (supply and exhaust) around the computing components, watertemperatures, and air and water flow rates, may be selected according tovarious acceptable design goals. Such parameters may be selected so thatcooling using cooling towers may provide all or substantially all of thecooling that is necessary for facility 102. The parameters may beenforced by a typical digital electronic control system.

Supplemental cooling may also be provided, such as by cooling plant 108a, including by providing commercial chillers with cooling plant 108 athat add to the cooling provided only by the cooling towers, asdiscussed in more detail below. The chillers may take a variety offorms, such as reciprocal chillers, air-cooled chillers, adsorptionchillers, and other appropriate forms. Because chillers generally takeappreciably more energy to operate than do simple evaporative coolingsystems such as cooling towers, operation of the chillers may be avoidedwhere possible. As a result, supplemental cooling systems such aschillers may be reserved for operation on very warm days or at othertimes when the load in facility 102 exceeds the ability to cool thefacility sufficiently with mere evaporative cooling. For example, such aload may exceed the available cooling capacity when one or more coolingplants are removed from operation, such as for maintenance.

Cooling plant 108 a is shown connected to supply header 104 and returnheader 106. The headers 104, 106 may take the form of large conduitssuch as steel pipes or other common and appropriate conduits forcarrying cooling fluid. The headers 104, 106 are shown for clarity hereas extending beyond one side of facility 102. A particular location androuting of the headers 104, 106 may also take various other formsaccording to the plan of facility 102 and the needs of the particularinstallation.

The headers 104, 106 are shown as being provided with a number of tapssuch as tap 110 and tap 112. The taps may take an appropriate form forproviding fluid communication between the headers 104, 106 and eithercooling units in facility 102 or cooling plants outside of facility 102.In particular, headers 104, 106 may be initially configured so as topermit the connection of multiple rows of cooling units and multiplecooling plants to the headers once those additional cooling units andcooling plants have been installed and are operational.

In addition, various isolation valves, such as valve 114, are shown onheaders 104, 106. The valves may be used to segment portions of theheaders 104, 106 so that portions on which construction is stilloccurring may be kept dry and not in fluid communication with the restof the system. Alternatively, valves such as shutoff valves or balancingvalves that can also act as shut-off isolation valves, may be installedon the taps, such as tap 110 and tap 112, to provide fluid-relatedsegmentation of the system. Such valves may permit uncapping of the tapsto connect additional components of the system, without providing fluidcommunication to the operational portions of the system (with subsequentopening of the valves to provide fluid communication when all componentshave been installed, sealed, and tested).

FIG. 1B is a plan view of the data center of FIG. 1A at a later stage ofconstruction. In this example, the data center 100 has been underconstruction for a period of time after that shown in FIG. 1A. Inparticular, groups of computers in the form of rows such as a row 116,have been installed inside facility 102. In this example, the datacenter computers are installed as back-to-back rows 118 of rack-mountedcomputers, where the computers back up to cooling units 119. Therack-mounted computers may, for example, take the form of a number ofcomputer motherboards, and possible carriers for supporting themotherboards, mounted vertically across a rack (e.g., in the form of aso-called blade server) or horizontally up and down a rack much likebread trays may be mounted in a bakers rack, or lunch trays in a rack.The fronts of the racks may be left open to the workspace insidefacility 102 and the backs of the racks may be left open to coolingunits 119 or may otherwise exhaust air out to cooling units 119 (e.g.,through fans mounted to each motherboard or otherwise associated witheach motherboard or group of motherboards).

In operation, then, circulating air may enter the fronts of the racksfrom the workspace, may pass across heated components in the racks, suchas microprocessors and memory chips, and may be exhausted out of theback of the racks into the cooling units 119, so that each cooling unitreceives heated exhaust air from racks on opposed sides. The coolingunits 119 may be provided with air circulating equipment such as fansand cooling equipment such as cooling coils, so that air is drawnthrough the coils and exhausted in a cooled state back into the datacenter workspace.

In one example, the cooling units are placed between the back edges ofthe computer racks, and exhaust cool air upward into an upper region ofthe workspace. In another implementation, air may enter a warm airplenum behind the computers and be drawn down into an under-floor space(e.g., below a real or false floor), where it may be gathered,circulated through cooling coils, and delivered back into the workspace,such as through perforated floor plates.

The cool air may then be circulated again to the front side of each rackand pulled back across the racks and into the cooling unit 119 again.Various other arrangements of computing machinery, and air circulatingand cooling equipment, may also be used.

The cooling units 119 may be individual units that are configured in arow and brought into close contact with the computing racks. In thisexample, two rows of computers have been installed in the facility 102,where each row contains two rows of computing racks sandwiched on eachside of corresponding cooling units. In one example, each rack may beapproximately five feet wide (having three bays of computers per rack),and six to eight feet, or 14 or more feet high.

In certain implementations, temporary filtration may be provided withthe racks. For example, if certain racks are installed and/or run duringconstruction, they may be subjected to dust, and filtration of the dustmay be preferable. In such a situation, large relatively flat sheets offiltration material may be formed and held against the fronts of theracks. For example, sheets of foam may be cut to approximate the heightof the racks (or at least of the air intake areas on the racks) and maybe pressed up against the racks to provide simple, low cost filtrations.Other filter media may be used in a similar manner, such as variousforms of sheets of material.

Such filters may be held in place in a variety of manners. For example,differences in air pressure may be enough to provide some connectionbetween the filters ad the fronts of the racks, such as to hold thebottoms of the filters against the racks if the tops of the filters areconnected. Greater strength of connection may be achieved with simpleclamps, adhesives, magnets attached to the filter media or a filterframe around the media, and other similar mechanisms. When constructionis complete or a need for the filtration otherwise ceases, thefiltration media may be removed and cleaned for re-use, or disposed of.Such configurations are discussed in more detail with respect to FIGS.7A and 7B.

A cooling loop 120, in the form of additional piping, has also beeninstalled and connected to headers 104, 106 in preparation forinstalling computing racks above the loop. Although vertical positioningis not discernible in this figure, the cooling loop 120 may be locatedbelow the level of the computer racks, as may cooling loops for theparticular computing racks that are shown here, where those additionalcooling loops are not visible in the figure. For example, the coolingloops may be located below a raised floor in a facility, or may belocated in a ceiling space of a level that is below the level where thecomputing racks are located. Similarly, the cooling loop may be locatedabove the computer units, though such location may raise issues withrespect to leaking water impinging on the computer racks. Cooling loop120 may also be located within or adjacent to cooling units 119.

Also in the figure, additional cooling plants 108 b-d have also beeninstalled with the data center 100. In the example, each cooling plant108 b-d is associated with a particular row of computer racks withinfacility 102. Such a correlation may be based on prior calculationsindicating that a cooling load for a row of racks is substantiallyequivalent to the cooling that may be supplied by a particular coolingplant 108 b-d. In a similar manner, if a cooling plant is able to coolsubstantially more than one row of computer racks, each cooling plantmay be associated with 2, 3, or more rows of computing racks. Also, anuneven number of cooling plants to computer racks may be used, such as1.5 cooling plants for two rows of computer racks, or one cooling plantmay cover a fraction of a row.

Each of the cooling plants 108 a-d is provided in a modular orsubstantially modular format. In particular, each cooling plant 108 maycontain within it each of the components or major components needed fora cooling plant, such as cooling towers, heat exchangers, valvingsystems, and pumps for cooling fluid. The fluid may take a variety offorms, including water, treated water, glycol mixtures, and otherfluids. In a like manner, each row or group of rows of computing racksmay also be modular, in that it does not share a cooling loop with otherrows or group of rows of computer racks. Control for the variouscomponents may be centralized, however, in various well-known manners.

Thermal storage systems, such as brine tanks or water tanks, may also beused so as to store cooling capacity during periods of lower coolingdemand, and to release that capacity during high load periods, so as tosupplement the core real-time cooling capacity of a system (e.g., fromcooling towers).

In this example, four cooling plants have been erected, and two rows ofcooling units and computer racks have been installed, with a third rowready to be installed. The next stage in construction may be to installadditional rows of cooling units and computer racks above cooling loop120 and to provide piping connections between cooling units for thosenew computer racks and the loop 120. After that, another cooling plantand another corresponding row of cooling units and computer racks may beinstalled. The process of adding cooling plants and rows of coolingunits and computer racks may continue until the facility is filled.

As shown, by using cooling plant 108 a that does not correspond to anyparticular load in facility 102, the data center 100 may always have oneadditional cooling plant compared to its nominal needs. This additionalplant provides additional capacity that permits the data center tooperate at full capacity (at least for the full capacity of the portionof the data center 100 that has been completed and commissioned) even ifone cooling plant breaks or needs to be brought down for maintenance.The additional plant also allows for additional capacity during highload periods when all the cooling plants are operational. In otherwords, where the load is n, the supply may be maintained under normalconditions and during construction at n+1 (or n plus another appropriatenumber). The additional cooling plants may be desirable to improvedreliability.

The headers 104, 106 may be sized so as to permit sufficient flow ofcooling fluid from one end of the headers 104, 106 to the other evenwhen one or more cooling plants are not operational. For instance,although in full operation, cooling fluid could flow readily from onecooling plant across to its associated group of cooling units in thefacility 102 so that the headers 104, 106 would need to carry verylittle fluid up-and-down in the figure, the headers 104, 106 may besized to allow flow of cooling fluid from one segment to the next (e.g.,up-and-down in the figure), such that if the top-most (in plan view)cooling plant fails, sufficient cooling fluid may be transferred upwardthrough the headers 104, 106 from the other cooling plants so as tocontinue full operation of the top-most group of cooling units.

FIG. 1C is a plan view of a data center during early construction thatis using leased cooling plant equipment 148. In general, theorganization of the data center is similar to that of the example datacenter 100 shown in FIG. 1A, but with the addition of the leasedequipment 148.

In particular, in this example the facility 140 is again organized to beserved by a number of cooling plants that will be installed and madeoperational while other portions of the facility 104 have already beencommissioned, and stay in operation or continue operation with minorinterruptions due to commissioning of additional portions of facility140. In the figure, one permanent cooling plant 145 has been connectedto a cooling fluid circulation system, such as in the form of a supplyheader 154 and a return header 156. Isolation shutoff valves 144 enabledthe cooling plant 145 to be fluidically disconnected from the rest ofthe circulation system, such as when maintenance or replacement ofcooling plant 145 is necessary.

In a like manner, leased equipment 148 is also connected to the fluidcirculation system and provides additional cooling capacity to thesystem. The leased equipment 148 may be connected via headers 150 andmay be isolated from the system by shutoff valves 146. The leasedequipment 148 may generally be smaller in size and capacity than thecooling plant 145, though larger equipment may also be in used inappropriate circumstances. For example, the leased equipment 148 may besized to be mounted and transported on a standard trailer from atractor-trailer combo. The leased equipment 148 may thus be quicklyconnected by flexible fittings and the like, and may be easilydisconnected and returned to a leasing company.

In one example process for initiating the operation of the data center,the main circulation system and the headers 150 may initially beinstalled. The leased equipment 148 may then be procured and attached tothe headers 150. Computing equipment may be installed inside thefacility 140, and may be commissioned to operate using cooling providedby the leased equipment 148. For example, a single group of computercooling units associated with a group of computer racks may be broughtonline and operated to provide partial operation of the facility 140.

While the limited operation is occurring, cooling plant 145 may beerected while shut-off valves 144 remain closed. When the cooling plant145 is complete and tested, shutoff valves 144 may be opened and thecooling plant 145 may be fluidically connected to the rest of the systemand brought on line, or commissioned. As the operation of the facility140 continues, additional computer cooling units and computer racks andcomputer cooling units may be installed and additional cooling plantsmay be added and connected to the fluid circulation system. Thoseadditional loads and cooling plants may be sequentially commissioned sothat the cooling fluid that is supplied generally matches the coolingload in a stepwise parallel manner, and as much of the facility as ispracticable may be brought online, or commissioned, immediately.

In some approaches, the cooling piping used by the temporary equipmentmay be removed when such equipment is removed, such as after one or morepermanent cooling systems is commissioned. Alternatively, the coolingpiping may be left in place so that temporary equipment may be broughtback on site and used whenever additional capacity is needed such asduring a particular warm spell or high demand period for computingservices, or during maintenance or unexpected break downs in thepermanent equipment.

In addition to cooling equipment, electrical equipment may also be usedon a temporary basis, while construction is occurring, for permanentelectrical equipment. For example, electrical substation 152 may beprovided in proximity to facility 140 and may provide a limitedelectrical service to the facility 140. For example, in a facilitydesigned to have 20 groups of computers, a leased substation designed tosupport 5% of the ultimate load may be provided on-site and may power afirst group of computers. In addition, a substation or other electricalequipment may be provided to power leased equipment 148 and otherancillary equipment for the facility 140, until sufficient permanentpower supply sources have been installed and commissioned.

The event of commissioning generally relates to a time when a system orsubsystem is brought into full operation. Generally, such commissioningrequires a review of a built system and an approval to begin fulloperation of the system. For example, commissioning may require approvalunder a particular organization's guidelines for approving operation ofdata centers. The organization may be the owner of the data centeritself, may be a third-party private organization (such as a ratingorganization), or may be a governmental organization such as a local orregional building department. Generally, commissioning of a data centermeans that the commissioned components are able to provide services to alarge portion of the intended end users of a system, such as to thepublic for an on line service provider. Pre-commissioning activitiesmay, in contrast, include testing of the computing components, testingof the cooling system components, and other similar testing orless-then-commercial activities.

FIG. 2 is a side sectional of a data center 200. In general, the datacenter 200 has a similar layout to the data center 100 shown in FIGS. 1Aand 2A. For example, the data center 200 generally includes a datacenter facility 202 in which computing resources are located, and one ormore associated cooling plants like plant 204 that provide cooling fluidto the data center facility 202. The particular arrangement of the datacenter facility 202 and cooling plant 204 may differ, and in thisexample, as in the other figures, the cooling plant 204 is being shownas a largely self-contained, modular cooling plant that is associatedwith one or more rows of computer racks 230.

Referring to components inside data center facility 202, computer racks230 are located on top of a floor, which may be a standard or raisedfloor, having a walking space or crawlspace below. For example, thebelow-floor space may be an area from about 1 foot in height to about 8feet or more in height. The below-floor area includes a supply fluidheader 218 and a return fluid header 220. In this example, the supplyfluid header 218 receives cooled water from cooling plant 204 and othersimilar cooling plants, while the return fluid header 220 receiveswarmed water back from the racks 230 and provides the warm water back tothe cooling plants, such as cooling plant 204. The facility 202 need nothave a raised floor, however and could be deployed on slab, withmechanical utilities above or below the slab.

Spurs 226 are provided off of each of the supply fluid header 218 andthe return fluid header 220, and extend linearly into the facility 202past (e.g., under, behind, or over) the computer racks 230. Atparticular locations, such as locations corresponding to individualcooling units associated with the racks 230, risers 228 extend upwardand connect to the cooling units. In this example, the cooling units arenot shown because they are located behind the racks 230 from theillustrated viewing angle. In general, the cooling units may includestandard cooling coils, along with air circulation fans that may assistin drawing air across computing components, perhaps in combination withfans located on particular computer motherboards or with particularcomputer motherboards, and cooling the air before re-circulating it intoworkspace 232 inside facility 202.

Cooling plant 204 contains a number of pieces of equipment needed toreceive, cool, and circulate fluid for facility 202. Primary coolingoccurs by cooling towers 206, which may take a variety of forms, such asupward flow open evaporative cooling towers. To prevent cooling towerwater from entering facility 202, such water, which may be driven bypump 208 and other pumps that are not shown, may be routed throughfluid-to-fluid heat exchanger 216. The exchanger 216 may take a varietyof forms, such as one or more plate heat exchangers or shell-in-tubeheat exchangers. Cooling water for facility 202 may be circulatedthrough the other side of heat exchanger 216 by supply pump 214 and/orreturn pump 212. The particular number of and form of pumps and valvesmay take various forms, and is shown here simply to illustrate possiblemanners in which to pipe such a cooling system.

Supplemental cooling may be provided by chiller 210, which may be in theform of an air-cooled chiller as discussed above and below, that may beoperated before cooling towers 206 are operational or before a waterpermit has been received. Generally, chiller 210 will be smaller incapacity than are the cooling towers 206.

As thus described, data center 200 may operate in a manner that matchesportions of the cooling load, such as by having rows of computer racksand associated cooling units, to particular cooling plants. In thismanner, each unit of cooling load may be brought on-line as acorresponding unit of cooling supply is made ready. The matching ofcooling load to cooling supply may be made less direct using coolingfluid mixing devices such as headers 218, 220, so that, although load ismatched with supply in a stepwise manner (when X BTUH's of load areadded, approximately X BTUH's of cooling may be added), particularincreases in load or supply may be taken up by supply from differentcooling plants across the system. In particular, a system may beprovided with greater diversity by allowing cooling water from one plantto be mixed with cooling water from an adjacent plant or another plant,such as by using common headers that permit lateral cooling fluid flowbetween modules.

FIG. 3 is a flowchart showing a process 300 for staged start up of adata center facility. In general, the process shows one example of atechnique for beginning operation of a data center using coolingequipment, and transitioning to higher capacity cooling equipment. Incertain circumstances, the lower capacity cooling equipment may bemaintained on site and made available for ready use, such as whensupplemental cooling is needed (e.g., on particular warm days or whenmain cooling units must be removed from service). Generally, thetemporary cooling equipment will have a higher cost of operation thanwill the higher capacity cooling equipment.

At box 302, air-cooled chillers are installed at a site. The air-cooledchillers may be sized to provide a portion of the cooling that willultimately be needed for a data center at the site. Some portions of theentire data center may be operated and be cooled by such air-cooledchillers.

Alternatively, or in addition, hybrid cooling towers (which may also betermed building heat exchangers) may also be installed. Such towers maybe operated in a non-evaporative mode, where make-up water would not beneeded, and in an open, evaporative mode, where make-up water would beneeded. The evaporative mode typically has much higher capacity andallows supply of acceptably cool fluid at higher outdoor dry-bulbtemperatures than does the non-evaporative mode, but uses substantiallymore water. Additional capacity with some increase in water usage may beobtained by misting water on the water conduits in the non-evaporativemode so that the conduits are cooled and more heat can be drawn out ofthe circulating water.

With such hybrid towers, the system may be operated initially (beforerelevant permitting or other go-ahead measures are obtained orperformed) in a non-evaporative mode, and then in an evaporative modewhen sufficient levels of make-up water may be obtained. In the initialphase, only a portion of a data center might be supported, which may beacceptable if only a portion of the data center has been constructed andinstalled. Additional loads may be installed in the data center while anorganization is waiting for a water permit or other appropriatepermitting, so that the loads (e.g., racks of computers) are ready tocome on-line when the higher capacity mode for the towers is ready tocome on line.

Also, in certain circumstances, provisions may be made to dispose ofwaste “blow down” water and other such water. For example, waste watertreatment facilities may be built or brought on site, such as intransportable modules, to provide such treatment capabilities, at leastuntil a waste water permit can be obtained.

At box 304, cooling towers may be installed at the site. The coolingtowers may be included as part of a modular unit with the air-cooledchillers, such as described above with respect to FIG. 2. For example, aframe may be erected or may be brought to the site such as on asemi-trailer, where the air-cooled chiller and associated pumps and heatexchangers are provided in the frame. The frame may be constructed so asto hold one or more cooling towers on its top edge, and the towers maybe installed after the equipment inside the frame is fully operational(because towers may generally be too tall for transport if stacked ontop of the frames during shipment). The cooling towers may also beconnected physically to the water circulation system but may be keptdisconnected fluidically, or may be kept closed in a non-evaporativemode, until approval is received to operate the towers.

At box 306, water supply and wastewater permits are obtained. With thepermit, the facility may now draw in make-up water from, for example, acity or regional water system or a well, to replace water thatevaporates in the towers. Such a permit may allow the facility tooperate at a much larger capacity, as the cooling towers may operatemuch more efficiently than may a chiller, and may thus be sized tohandle a much larger load than would be economical with the chiller.Other appropriate triggering mechanisms for starting the operation ofthe cooling towers may also be used, such as a building inspectorapproval or other similar approvals.

At box 308, the cooling towers are connected fluidically to the coolingsystem, or are opened for evaporative cooling. The towers may havepreviously been connected physically to the system, but shut-off valvesmay have remained closed until the towers could provide water to thesystem. Upon adding the cooling towers to the system, the chiller may bepowered down and taken off-line, and the cooling system switchedprimarily to cooling tower cooling, as shown at box 310. At later times,the chiller may be added back into the system, such as on particularlywarm days or when heat loads in the facility are very high or water isnot available for the cooling towers.

FIG. 4 is a flowchart of a process 400 for transitioning a data centerfrom temporary to permanent utility equipment. In general, the process400 involves leasing cooling equipment that can handle a portion of theload of a full facility, using the leased cooling equipment to get thefacility operational quickly, building permanent equipment while thefacility is run at a reduced level, and transitioning to the permanentequipment and raising the capacity of the facility when the permanentequipment is ready. Such transitioning may occur in a single step, suchas bringing on the permanent equipment at one time, or may occursequentially in a number of steps, such as by providing a number ofdiscrete cooling plants for the facility and bringing them on-linesingularly or in smaller subgroups.

At box 402, cooling equipment for the facility is leased. Leasing theequipment may permit a facility to acquire equipment more quickly thanpurchasing and installing equipment. However, leased equipment isgenerally smaller and of lower capacity than would be permanentequipment. For instance, leased equipment may be of a size that may betransported easily on semi-trailer trucks or by other similarmechanisms. Thus, in this example, the leased equipment is not capableof serving the entirety or a substantial entirety of the facility.Rather, the leased equipment is used to provide quick up time for aportion of the facility—in this example, a data center.

At box 404, the leased cooling equipment is connected to the datacenter. In one example, a common circulation loop, such as in the formof a supply header and return header, may be provided for a facility,where the loop may be segmented by shutoff valves located in the loop orin taps off of the loop, so that connections may be made to the loopwith new equipment while existing equipment is still operating in theloop.

At box 406, a first phase of the data center is commissioned. Forexample, an initial group of computers and associated computer coolingunits inside the facility may be installed and made operational, as maya group of leased cooling plants for serving such units. In this step,the cooling units and computing units and the associated leasedequipment may be commissioned and brought online.

At box 408, initial purchased equipment is installed. For example, afirst permanent cooling plant may be added to the facility and connectedto the cooling system. The connection may be physical but not fluidicinitially, in that all necessary piping may be connected, but shutoffvalves may isolate the new equipment from the rest of the system.

At box 410, the initial purchased equipment is commissioned such that itis fluidically connected to the cooling system. If the initial purchasedequipment is sufficient to provide cooling for the currently operationalportions of the facility, some or all of the leased equipment may bedecommissioned and removed, as shown at box 412. For example, the leasedequipment may be shut down, isolated fluidically from the rest of thesystem using shutoff valves, and returned to a leasing company.

At box 414, the remaining purchased equipment for a facility may beinstalled and commissioned. For example, the initial purchased equipmentmay be sufficient to supply the then-existing portion of a facility, butinsufficient to supply all of the cooling loads that ultimately may beinstalled in the facility. As a result, after the leased equipment isremoved, construction may continue to add computer cooling units andcomputers, and corresponding construction may continue to add coolingplants for the facility. Each cooling plant may be largelyself-contained and connected to a common circulation loop using taps andshutoff valves as described above.

A particular number of cooling plants and groups of computers may varywith each application, and may be brought online, or commissioned, invarious manners. In one example, the commissioning may be sequentialfrom one end of a facility to another end of the facility, with coolingloads and cooling plants being located one after the other adjacent toeach other in a line. Such an arrangement is shown, for example, in FIG.1B.

FIG. 5 is a flowchart of a process for sequential commissioning of partsof a data center system. In general, the process 500 shows theconstruction of a data center with a cooling system in a sequentialmanner so that an initial portion of the system may come on-linefollowed by additional portions of the system. In this example, oneextra cooling plant is commissioned in advance to provide additionalcooling in the event that such cooling is needed or that another coolingplant is removed from service.

At box 502, common cooling system components that connect multipleportions of the system are installed. For example, headers in the formof conduits that serve an entire facility can be installed and can besegmented so that separate groups of cooling units can be connectedindividually to the common components. The segmentation may occur, forexample, by providing valves in the headers themselves, or in taps offof the headers that serve groups of cooling units. All of the commoncomponents may be installed initially, or portions may be installed. Forexample, headers may be installed to serve half a facility, withshut-off valves in taps and a single shut-off valve in each header nearthe header ends. The headers may then be extended after the first halfof the project is completed, by adding conduits to the ends of theheaders and opening the shut-off valves.

At box 503, common utility system components are installed. Suchcomponents are similar to the common cooling system components, andcould be thought of as a single group of components. In this example,however, the utility system components are upstream of the coolingsystem components and provide utilities to such components. For example,water supply piping to a site and main electrical service infrastructuremay be though of as common utility system components. Such componentsthat are needed to serve an entire facility or large portions of thefacility, may be installed and commissioned initially so as to minimizedisruptions to utility services as modular portions of a facility areadded sequentially to the system.

At box 504, an “extra” cooling plant or plants is installed andcommissioned. This “extra” plant may not be associated with anyparticular group of cooling units, and may simply serve to provideback-up or additional reserve cooling capacity in case it is needed. Thepresence of a single cooling plant that is matched to the cooling loadmay be sufficient to commission that portion of the facility (and suchportion may be commissioned), or it may be insufficient (in which case,commissioning will need to wait). In other words, where the demand isfor n plants, the construction may be phased so that n+1 plants arealways available. With the cooling capacity ready, the extra servers orother cooling loads may be commissioned once they have been installed(box 505). (Installation of the servers can begin before the extracooling capacity is ready.)

At box 506, an iterative process of expanding the facility capacity isbegun. In particular, an additional cooling plant or cooling plants areinstalled and commissioned. Likewise, at box 508, additional groups ofcomputer cooling units (along with associated computer racks containingoperating computers and other computer equipment like storage andnetworking devices) are installed and commissioned. Such cooling plantsand groups of cooling units may be arranged in a manner like that shownin FIG. 1B so that there is a 1-to-1 or x-to-1 relationship betweengroups of cooling units (e.g., in linear rows) and cooling plants. Also,particular cooling plants and associated cooling loads may be added tothe system simultaneously or at different times. For example, a coolingplant may first be added and commissioned so that the system has excesscapacity (even beyond that provided by the “extra” plant), and then theadditional load may be added when the new plant is proven and stable.Alternatively, the load and a corresponding plant may be brought uptogether so that there is only one major change made to the overallsystem for each additional unit of load and supply. In addition, extrautilities may be installed and commissioned where they are needed. (Box508).

Certain size restrictions may also be taken into account in determiningthe correlation between cooling loads and cooling plants. For example,if each row of computer racks, cooling units, and space for workersrequires 10 feet, and each cooling plant is about 20 feet wide, theneach cooling plant may be sized to service two rows of cooling units.

At box 510, a decision is made to determine whether all of the plannedplants and cooling units have been installed (e.g., is the facilitycomplete). If they have, then the process ends (box 512) and if theyhave not, the iterative part of the process 500 repeats.

FIG. 6A is a plan view of a data center 600 during construction, showingcomputers with cooling units and a cooling piping system. The datacenter 600 is generally made up of a computing facility 602 that mayhouse thousands or tens of thousands of computing systems, and coolingplants 608 a-f to provide cooling for those systems. In particular,groups of computers in the form of rows such as a row 616, have beeninstalled inside facility 602.

In this example, four rows of computers 616 have been installed in thefacility 602, where each row contains two rows of computing rackssandwiched on each side of corresponding cooling units 619. In oneexample, each rack may be approximately five feet wide (having threebays of computers per rack), six to eight feet or more high, and about1.5 feet deep. In this example, the data center computers are installedas back-to-back rows of rack-mounted computers, where the computers backup to cooling units.

One utility that is provided to the facility 602 is in the form ofcooling water, supplied by a cooling plants 608 a-f. The cooling watermay generally be circulated to cooling units located throughout thefacility 602, such as water-to-air heat exchangers for cooling air inthe facility 602 that has been heated by computing systems and computersin the facility at 602.

Cooling plants 608 a-f are shown connected to supply header 604 andreturn header 606. A number of cooling loops 620, in the form ofadditional piping, have also been installed and connected to headers604, 606 in preparation for installing computing racks above the loops620. Although vertical positioning is not discernible in this figure,the cooling loop 620 may be located above, behind, below the level ofthe computer racks, as may cooling loops for the particular computingracks that are shown here, where those additional cooling loops are notvisible in the figure. For example, the cooling loops may be locatedbelow a raised floor in a facility, or may be located in a ceiling spaceof a floor below a floor where the computing racks are located.

While the data center is in operation, circulating air may enter thefronts of the racks from the workspace, may pass across heatedcomponents in the racks, such as microprocessors and memory chips, andmay be exhausted out of the back of the racks into the cooling units619, so that each cooling unit receives heated exhaust air from racks onopposed sides. The cooling units 619 may be provided with aircirculating equipment such as fans and cooling equipment such as coolingcoils, so that air is drawn through the coils and exhausted in a cooledstate back into the data center workspace. Cooled coolant fluid(generally water) is supplied to the cooling units 619 by cooling loops620, and the warmed coolant fluid is taken away in cooling loops 620 andreturned to cooling plants 608 a-f for cooling.

In addition, various isolation valves, such as valve 614, are shown onheaders 604, 606. The valves may be used to segment portions of theheaders 604, 606 so that portions of facility 602 in which constructionis still occurring may be kept dry and not in fluid communication withthe rest of the system. Alternatively, valves, such as shutoff valves orbalancing valves that can also act as shut-off valves, may providefluid-related segmentation of the system. Such valves may permituncapping of the taps to connect additional components of the system,without providing fluid communication to the operational portions of thesystem. The valves 614 may also be used to permit one or more coolingplants to supply fluid to multiple cooling loops during initialconstruction of the computing facility 602, when not all computer unitshave been installed.

The cooling loops 620 also include isolation valves, such as valve 630,that allow unneeded portion of the cooling loop 620 to not be used untilneeded. The cooling loops 620 also include taps, such as tap 631, thatallow for connection to a cooling unit once a cooling unit is ready forinstallation. The placement of isolation valves and taps in the coolingloops allow for construction of the computing facility in stages, andenable the computer servers to be built in rows in the computingfacility as shown in FIG. 6A.

In the pictured example, each cooling plant 608 b-f is associated with aparticular cooling loop within facility 602. As shown, FIG. 6Aillustrates one example of a data center using an interleaved coolingapproach. Thus, the cooling loops 620 are generally placed in aperpendicular configuration compared to the rows of computing units 616in the facility. In such a configuration, the cooling supplied by aparticular loop may be provided to units in separate rows. As a result,if one loop needs to be taken off-line, the effect of such a change canbe spread more easily across the facility, where other loops servingadjacent computers in various rows may pick up most of the slack.

Also, warmed air may circulate up and down each row behind the racks618. As a result, if cooling units dispersed throughout a row are notcurrently receiving cooling water because an associated loop is down forservicing, the effect of that problem can be accommodated by adjoiningunits. Moreover, where the cooled air is discharged into the generalworkspace of the facility 602, any remaining un-cooled air willgenerally blend with the cooled air before it is re-circulated throughthe computer racks.

Other utilities may also be connected to the facility 602. Suchutilities can include connections to the internet, connections to wastewater systems (temporary or permanent), connections to power (temporaryor permanent facilities), and other similar utilities needed to operatethe data center 600.

FIG. 6B is a plan view of a data center 650 cooling piping system. Thedata center 650 is similar to other data centers discussedabove—generally made up of a computing facility 652 that may housethousands or tens of thousands of computing systems, and a number ofcooling plants 658 a-d connected to cooling loops 670 in the facility652 via supply and return cooling fluid headers 654, 656. In thisexample, cooling loops 670 in the facility 652 run parallel with rows ofcomputers 669 overhead (and shown in dashed lines here).

Take-offs 680, 682 from the loops 670, however, are run to multipledifferent rows of computers 669. (Where the take-offs are shown in thefigure crossing part of a loop, the two are not connected in thisexample.) In particular, one loop may serve a row on one side of theloop and then a row on the other side of the loop. An adjacent coolingfluid loop may likewise serve cooling units in any of the rows that arebetween the loops. In this manner, adjacent cooling units in a row ofcomputers may be served by different loops. Thus, if one loop is takenout of service, only some of the cooling units in a row will go down.Where air from adjacent units can be mixed and taken up by other units,the effect of taking down one loop can be spread out through the systemand no row will need to be deactivated during the maintenance period.Also, where warm air and cool air is exhausted into the main workspaceof facility 652 from adjacent units, the workspace may serve effectivelyas a thermal capacitor, so that the warm and cool air mix before beingre-circulated through the computer racks and the ultimate rise intemperature of the mixed air will be negligible. Any rise in temperaturemay also be offset by increasing the flow of cooling fluid through thestill-operating loops or decreasing its temperature slightly.

Such an arrangement may also work well where sequential commissioning ofa data center is employed. In particular, crossing of cooling fluid tapsfrom one row to another may be conducted in discrete subsets of rows,such as two rows or four rows. The sequential commissioning may thenoccur according to those sub-set units, such as where a pair of loopsserving four rows is installed and the rows and loops are commissionedtogether (perhaps along with an appropriately-sized cooling plant).

Also, shut off valves may be provided on the taps to each cooling unit(not shown) so that single rows of units may be commissioned even wherea loop serves multiple rows. For example, the bottom loop illustrated inFIG. 6B serves four different rows of racks. The loop may be installedinitially and stubs having shut off valves may be installed at that timealso. The stubs may be extended to the first row of racks (at the bottomin the figure), and the loop may be fluidically connected to the coolingsystem when the first row is commissioned. For example, only theleft-most stub on the lowest loop in the figure may be fully built outinitially, with the rest of the stubs left as short stubs.

Additional piping may be added as additional rows of racks are added,For example, When the second row of computer racks is installed, thetaps for that row may be extended and connected, and the shut-off valvesserving stubs to that row may be opened when that row is commissioned.In the pictured example, the third (top) loop serves the second row ofcomputers form the bottom; in such a situation, the loop may not yet beinstalled when the second row of computers from the bottom is ready forcommissioning. As a result, the particular cooling that was to be servedby the top loop may remain unserved until the final loop iscommissioned. Although such an arrangement may create a hot spot in thesystem, the warm air coming out of that unit will be readily blendedwith cool air coming from adjacent units that are being served by thefirst and second loops, via cold-air mixing, which may provideadditional diversity in a system. Thus, in these various manners greaterdiversity may be provided to a cooling system, particularly on the waterside, so that adequate cooling may be provided even when a significantportion of the system is inoperable.

FIG. 7A is a plan view of a data center showing continued operation ofcomputers while additional computers are installed, while FIG. 7B is asection view of the data center in FIG. 7A. The data center 700 isgenerally made up of a computing facility 702 that may house thousandsor tens of thousands of computing systems. The particular dimensions ofthe racks in relation to the size of the facility is exaggerated herefor purposes of clarity

In FIG. 7A, data center equipment has been placed in rows 719 andlocated in the data center 700 to fit efficiently with the given supportcolumn layout of the computing facility 702. In the figure, computingrows 716 have been spaced so as to fit around the support columns 704shown in FIG. 7A. Computing rows 716 are composed of a row of coolingunits 719 between the two rows of racks 718. The cooling units 719 aremodular and approximately 5-10 feet in length so that they can be movedeasily, e.g., with a forklift, and so that they can be spaced apart fromeach other so as to be positioned around the columns and thus make thecolumns effectively disappear in the data center.

Access rows 710, 712, 714 for accessing the computer racks have beenpositioned so as to fit between the computer rack rows 716 that includethe support columns. The access rows 710, 712, 714 may be sized so as topermit technicians in a facility to service machines held in racks inthe facility, and to install and remove the racks, such as when theracks are mounted in wheeled chassis.

As a result of locating the rows on the columns, rather than picking aset width for each item, certain access rows may be wider than areothers. In addition, cooling modules in the row of cooling units 719having support columns may also be unevenly spaced. Such a reduction incooling modules may have no effect, if the computers that exhaust airinto those rows are not generating more heat than even the reducednumber of modules can absorb. Because warm air can move up and down arow, and thus mix between units, additional diversity may be created inthe system. Even if there is some effect, the large area in thecomputing facility 702 helps serve as an absorber, or capacitor, toprevent substantial temperature variation or disturbances from having anappreciable effect on the operation of any computer rack in the facility702.

The computers in rows such as row 716 are in operation but the facility702 is still under construction, so an isolation curtain 725 divides thecommissioned portion of the facility 702 from the portion still underconstruction. The isolation curtain 725 is installed to reduceenvironmental contamination or effects from the ongoing construction andinstallation of the facility from reaching the operating portions of thecomputing facility 702. The curtain may take the form of a flexiblematerial such as plastic or tarpaulin sealed around the perimeter of thefacility 702 and extended across an entire cross-section of the facility702.

In certain implementations, temporary filtration may also be providedwith the racks. For example, as portions of the computing facility 702are operating while construction and installation proceeds in otherareas of the computing facility, the rack rows 718 may be subject todust, and other contaminants. Local filtration at the racks maytherefore be beneficial in operating the computing facility 702, evenwhen an isolation curtain 725 is already in place. Such filtration mayresult in a pressure drop, however, and thus may require additionalpower from the fans providing air circulation, resulting in higherelectrical costs and perhaps in larger problems balancing pressuredifferences in the facility 702. Because such filtration may not beneeded after construction and a sufficient break-in period are complete,temporary filtration may be beneficial.

FIG. 7B depicts the provision of such air filtration. To provide suchair filtration, large relatively flat sheets or rolls of filtrationmaterial 730 may be installed and held against the fronts of the racks718. For example, sheets of open-cell foam may be cut to approximate theheight of the racks (or at least of the air intake areas on the racks)and may be pressed up against the racks to provide simple, low-costfiltration. Alternatively, rolls of foam material may be unrolled andplaced against the fronts of the racks to act as a filtration material.Other filter media may be used in a similar manner.

Such filters may be held in place in a variety of manners. For example,differences in air pressure may be enough to provide some connectionbetween the filters and the fronts of the racks, such as to hold thebottoms of the filters against the racks if the tops of the filters areconnected. Greater strength of connection may be achieved with simpleclamps, adhesives, or magnets attached to the filter media or a filterframe around the media, and other similar mechanisms. When constructionis complete or a need for the filtration otherwise ceases, thefiltration media may be removed and cleaned for re-use or discarded.

FIG. 8 is a conceptual diagram showing the use of portable utilityservices to serve a data center temporarily. Such an approach allows forthe operation of a rapid deployment data center that can be in operationmuch more quickly than a data center that uses traditional, permanentcomponents from the start.

The rapid deployment center includes a number of transportable modules.Because the modules may be transported, installed, operated,uninstalled, and transported again, they may be referred to as mobilemodules. The modules may be transported by truck, rail, ship, air, orother method of transportation. The modules may be transported to adesired location and installed for use at that location. The modules maybe used to create a temporary data center, to provide computing capacitywhile installation of a permanent data center proceeds, to augment analready-existing data center, to temporarily support an existing datacenter (such as during maintenance, emergencies, etc.), or for otheruses. The modules may be of any type that would be used in a datacenter.

The mobile modules may be selected based on whatever needs of a datacenter are part of the critical path of a project plan to have the datacenter operations. Therefore, mobile or transportable modules mayinclude data processing modules, power modules, cooling modules, watersupply modules, waste water modules, data transmission modules, datacenter control modules, shelter modules (such as a tent, prefab buildingmaterials, etc.) and other types of modules that may be useful orneeded. For example, where a facility has not yet received a waterpermit, make-up water for evaporation in cooling towers or for mistingcooling towers may be provided.

A number of transportable modules may be used in a rapid deployment datacenter, as building blocks for such a center. In one example, a numberof mobile modules may be identified and transported together as a mobilegroup 810 to create a rapid deployment data center. The modules may belocated within shipping containers (where feasible) or may be inspecific containers depending on the type of module. Packaging of amodule in a shipping container may permit for more flexible andautomated data center transportation and assembly.

The mobile group 810 may be transported and installed at a data centerlocation 890. All of the needed modules may be transported to the datacenter location 890 from one or more origination points. The variouscomponents in the modules may be assembled efficiently at their pointsof origin, where applicable. Alternatively, some modules may be shippedfrom some distance, while others such as water trucks, may be obtainedor leased more locally to the data center location. The data centerlocation 890 may be prepared before or after the mobile group 810arrives at the location 890.

The location 890 may be in a variety of states for establishing a datacenter location. For example, the data center location 890 may be acleared and leveled area, a graveled area, an asphalt or concrete lot orpad, or may be another type of ground condition amenable to installationof the rapid deployment data center modules. The location may be onelarge location, or may be separated into a number of sub-locations (suchas two separate parking lots, etc.) The data center location 890 may beopen, may be within a warehouse or other building, may be located withina temporary building or shelter installed before or after arrival of themobile group 810, or may be located within a tent installed before orafter arrival of the mobile group 810.

FIG. 8 shows one example of a rapid deployment data center 815 installedand operational at data center location 890. Rapid deployment datacenter 815 includes two mobile data processing modules 820, 825 thathave been transported and installed. The data processing modules includeracks of servers, and are described in more detail in regards to FIGS.9A and 9B.

The mobile data processing modules may be designed to operateindependently of each other. Alternatively, the mobile data processingmodules 820, 825 may be designed and constructed such that the sidewallsand/or end walls of the containers are removable and a connection toanother data processing module may be made. This design feature allowsfor multiple data processing modules to be installed adjacent to eachother to form a single connected data processing facility, and enablescommon access to all of the servers within the facility, even thoughthey may be located in different modules. In addition to the removablesidewalls and/or end walls, the design may include elements to enable agood connection with adjacent data centers, such that a secure andcontained physical environment may be created. Thus, the common facilityfeature further allows for common air circulation to create a morestable and common environment within the data processing modules.Furthermore, such a design may further allow for easier utilityconnections, as utilities in the modules can be connected together, andtherefore may only require one external connection for each utility tothe common facility, rather than multiple connections to multipleseparate data processing modules. In other designs, some utilities maybe connected together and supplied in one location to the commonfacility, while other utilities may continue to be supplied to eachindividual data processing module.

Rapid deployment data center 815 includes a number of other modules insupport of the data processing modules 820, 825. A mobile powergeneration module 840 has been transported and installed at the site890, and is connected to the data processing modules 820, 825 thoughcables 870. A mobile water cooling tower 850 has been installed at thesite 890, and is connected to the data processing modules 820, 825through piping 872 and 873. Cooled cooling fluid (typically water) issupplied from the cooling tower 850 through a supply pipe 872, where itis used to cool the air circulated through the data processing modules820, 825. Warmed cooling fluid then returns to the cooling tower 850through return piping 873, where the cooling fluid is cooled to bereturned to the data processing modules.

The cooling fluid is cooled at the cooling tower by using one or moreheat exchangers to exchange heat from the cooling fluid and water at thecooling tower 850. Accordingly, make-up water is supplied to the coolingtower 850 by water piping 874 from a water supply module 830 (shown as atanker truck). Water is circulated, and evaporated in order to cool thewater for exchanging heat with the cooling fluid in heat exchangerslocated within the cooling tower 850. In order to continue operatingeffectively, water must be drawn from the cooling tower. Over time, thecirculated water typically includes elevated levels of sediments,chemicals, or other materials. Therefore, portions of the circulatedwater may be drawn to a waste water module 835 by waste water piping876. Typically, a waste water module 835 will be used in areas where asewer connection cannot be made, or where regulations so requiredisposal of sediment and other materials. Although a cooling tower 850is shown as the cooling module, other options may also be used. Forexample, an air cooler, air conditioner, hybrid cooling tower, chillers,or other options may be used.

Rapid deployment data center 815 also includes a connection to theinternet 860 over optical fibers 878 or other transmission hardware. Ingeneral, any appropriate type of connection may be made, including byusing optical fiber, wiring, a wireless connection such as a cellconnection or satellite, or other type of connection using anytransmission equipment. Thus, the rapid deployment data center 815 mayalso include one or more transmission equipment modules, such as a celltower, satellite linkage, or other equipment.

In some cases, the rapid deployment data center 815 may also include acontrol module, which may be used to monitor and control utilities,transmissions, and other necessary items at the location 890. In othercases, not all modules shown in FIG. 8 may be included in the rapiddeployment data center. For example, if a location has connections towater supply and waste water treatment, the water supply and waste watermodules would not be included in the rapid deployment data center atthat location. As another example, if a location has access to power,but lacks other services, a rapid deployment data center at thatlocation would not include a power generation module.

The rapid deployment data center 815 may be used in various manners. Forexample, it may serve as a temporary data center that is dismantled allat once after some time, such as when a permanent data center has beencommissioned and is operational. In other cases, the rapid deploymentdata center 815 may serve as a foundation for constructing a permanentdata center, and modules may be individually dismantled over time aspermanent facilities are constructed or permanent connections made toreplace the various modules. In other cases, a small mobile data centermay provide services for some time, such as when the need for apermanent facility is not yet great enough to create a sufficient needfor a permanent facility. Also, more permanent components may be used insome portions of the data center, such as for computer racks andassociated cooling equipment inside a data center facility, whileportable modules may be used outside the facility, such as to providemake-up water or remove sediment and other waste.

After use, the rapid deployment data center may be decommissioned andremoved. As discussed, the decommissioning may occur over a period oftime or may happen all at one time. Generally, the modules may bedecommissioned and sent to a location for storage, sent to a location tobe refurbished or repaired, shipped to a new location, returned to aleasing company, sold, or other disposition.

FIGS. 9A and 9B show plan and sectional views, respectively, of amodular data center system. The system may include one of more dataprocessing centers 900 in shipping containers 902. Although not shown toscale in the figure, each shipping container 902 may be approximately 40feet along, 8 feet wide, and 9.5 feet tall (e.g., a 1AAA shippingcontainer). In other implementations, the shipping container can havedifferent dimensions (e.g., the shipping container can be a 1CC shippingcontainer). Such containers may be employed as part of a rapiddeployment data center like that discussed with respect to FIG. 8.

Each container 902 includes side panels that are designed to be removed.Each container 902 also includes equipment designed to enable thecontainer to be fully connected with an adjacent container. Suchconnections enable common access to the equipment in multiple attachedcontainers, a common environment, and an enclosed environmental space.

Each container 902 may include vestibules 904, 906 at each end of therelevant container 902. When multiple containers are connected to eachother, these vestibules provide access across the containers. One ormore patch panels or other networking components to permit for theoperation of data processing center 900 may also be located investibules 904, 906. In addition, vestibules 904, 906 may containconnections and controls for the shipping container. For example,cooling pipes (e.g., from heat exchangers that provide cooling waterthat has been cooled by water supplied from a source of cooling such asa cooling tower) may pass through the end walls of a container, and maybe provided with shut-off valves in the vestibules 904, 906 to permitfor simplified connection of the data center to, for example, coolingwater piping. Also, switching equipment may be located in the vestibules904, 906 to control equipment in the container 902. The vestibules 904,906 may also include connections and controls for attaching multiplecontainers 902 together. As one example, the connections may enable asingle external cooling water connection, while the internal coolinglines are attached together via connections accessible in vestibules904, 906. Other utilities may be linkable in the same manner.

Central workspaces 908 may be defined down the middle of shippingcontainers 902 as aisles in which engineers, technicians, and otherworkers may move when maintaining and monitoring the data processingcenter 900. For example, workspaces 908 may provide room in whichworkers may remove trays from racks and replace them with new trays. Ingeneral, each workspace 908 is sized to permit for free movement byworkers and to permit manipulation of the various components in dataprocessing center 900, including providing space to slide trays out oftheir racks comfortably. When multiple containers 902 are joined, theworkspaces 908 may generally be accessed from vestibules 904, 906.

A number of racks such as rack 919 may be arrayed on each side of aworkspace 908. Each rack may hold several dozen trays, like tray 920, onwhich are mounted various computer components. The trays may simply beheld into position on ledges in each rack, and may be stacked one overthe other. Individual trays may be removed from a rack, or an entirerack may be moved into a workspace 908.

The racks may be arranged into a number of bays such as bay 918. In thefigure, each bay includes six racks and may be approximately 8 feetwide. The container 902 includes four bays on each side of eachworkspace 908. Space may be provided between adjacent bays to provideaccess between the bays, and to provide space for mounting controls orother components associated with each bay. Various other arrangementsfor racks and bays may also be employed as appropriate.

Warm air plenums 910, 914 are located behind the racks and along theexterior walls of the shipping container 902. A larger joint warm airplenum 912 is formed where the two shipping containers are connected.The warm air plenums receive air that has been pulled over trays, suchas tray 920, from workspace 908. The air movement may be created by fanslocated on the racks, in the floor, or in other locations. For example,if fans are located on the trays and each of the fans on the associatedtrays is controlled to exhaust air at one temperature, such as 40° C.,42.5° C., 45° C., 47.5° C., 50° C., 52.5° C., 55° C., or 57.5° C., theair in plenums 910, 912, 914 will generally be a single temperature oralmost a single temperature. As a result, there may be little need forblending or mixing of air in warm air plenums 910, 912, 914.Alternatively, if fans in the floor are used, there will be a greaterdegree temperature variation from air flowing over the racks, andgreater degree of mingling of air in the plenums 910, 912, 914 to helpmaintain a consistent temperature profile.

FIG. 9B shows a sectional view of the data center from FIG. 9A. Thisfigure more clearly shows the relationship and airflow betweenworkspaces 908 and warm air plenums 910, 912, 914. In particular, air isdrawn across trays, such as tray 920, by fans at the back of the trays919. Although individual fans associated with single trays or a smallnumber of trays, other arrangements of fans may also be provided. Forexample, larger fans or blowers, may be provided to serve more than onetray, to serve a rack or group or racks, or may be installed in thefloor, in the plenum space, or other location.

Air may be drawn out of warm air plenums 910, 912, 914 by fans 922, 924,926, 928. Fans 922, 924, 926, 928 may take various forms. In oneexemplary embodiment, the may be in the form of a number of squirrelcage fans. The fans may be located along the length of container 902,and below the racks, as shown in FIG. 9B. A number of fans may beassociated with each fan motor, so that groups of fans may be swappedout if there is a failure of a motor or fan.

An elevated floor 930 may be provided at or near the bottom of theracks, on which workers in workspaces 908 may stand. The elevated floor930 may be formed of a perforated material, of a grating, or of meshmaterial that permits air from fans 922, 924 to flow into workspaces908. Various forms of industrial flooring and platform materials may beused to produce a suitable floor that has low pressure losses.

Fans 922, 924, 926, 928 may blow heated air from warm air plenums 910,912, 914 through cooling coils 962, 964, 966, 968. The cooling coils maybe sized using well known techniques, and may be standard coils in theform of air-to-water heat exchangers providing a low air pressure drop,such as a 0.5 inch pressure drop. Cooling water may be provided to thecooling coils at a temperature, for example, of 10, 15, or 20 degreesCelsius, and may be returned from cooling coils at a temperature of 20,25, 30, 35, or 40 degrees Celsius. In other implementations, coolingwater may be supplied at 15, 10, or 20 degrees Celsius, and may bereturned at temperatures of about 25 degrees Celsius, 30 degreesCelsius, 35 degrees Celsius, 45 degrees Celsius, 50 degrees Celsius, orhigher temperatures. The position of the fans 922, 924, 926, 928 and thecoils 962, 964, 966, 968 may also be reversed, so as to give easieraccess to the fans for maintenance and replacement. In such anarrangement, the fans will draw air through the cooling coils.

The particular supply and return temperatures may be selected as aparameter or boundary condition for the system, or may be a variablethat depends on other parameters of the system. Likewise, the supply orreturn temperature may be monitored and used as a control input for thesystem, or may be left to range freely as a dependent variable of otherparameters in the system. For example, the temperature in workspaces 908may be set, as may the temperature of air entering plenums 910, 912,914. The flow rate of cooling water and/or the temperature of thecooling water may then vary based on the amount of cooling needed tomaintain those set temperatures.

The particular positioning of components in shipping container 902 maybe altered to meet particular needs. For example, the location of fansand cooling coils may be changed to provide for fewer changes in thedirection of airflow or to grant easier access for maintenance, such asto clean or replace coils or fan motors. Appropriate techniques may alsobe used to lessen the noise created in workspace 908 by fans. Forexample, placing coils in front of the fans may help to deaden noisecreated by the fans. Also, selection of materials and the layout ofcomponents may be made to lessen pressure drop so as to permit forquieter operation of fans, including by permitting lower rotationalspeeds of the fans. The equipment may also be positioned to enable easyaccess to connect one container to another, and also to disconnect themlater. Utilities and other services may also be positioned to enableeasy access and connections between containers 902.

Airflow in warm air plenums 910, 912, 914 may be controlled via pressuresensors. For example, the fans may be controlled so that the pressure inwarm air plenums is roughly equal to the pressure in workspaces 908.Taps for the pressure sensors may be placed in any appropriate locationfor approximating a pressure differential across the trays 920. Forexample, one tap may be placed in a central portion of plenum 912, whileanother may be placed on the workspace 908 side of a wall separatingplenum 912 from workspace 908. For example the sensors may be operatedin a conventional manner with a control system to control the operationof fans 922, 924, 926, 928. One sensor may be provided in each plenum,and the fans for a plenum or a portion of a plenum may be ganged on asingle control point.

For operations, the system may better isolate problems in one area fromother components. For instance, if a particular rack has trays that areoutputting very warm air, such action will not affect a pressure sensorin the plenum (even if the fans on the rack are running at high speed)because pressure differences quickly dissipate, and the air will bedrawn out of the plenum with other cooler air. The air of varyingtemperature will ultimately be mixed adequately in the plenum, in aworkspace, or in an area between the plenum and the workspace.

FIG. 10 shows a schematic diagram of a hybrid cooling tower system 1075for use with a data center (not shown). In general, the system 1075 canoperate in two modes: a non-evaporative mode that provides limitedcooling capacity but also requires very little or no make-up water, andan evaporative mode that provides much greater capacity but also usesmuch more make-up water.

The pictured system 1075 takes warm coolant that is supplied from a datacenter via piping 1013 to be cooled by the cooling tower 1020 and thensupplied back to the facility via piping 1015. System 1075 may includevarious components needed to provide cooled water or other fluids to thecomputer racks in the data center. In some implementations, the system1075 may be of a simplified design that includes an operating portionthat contains many of the pumps and heat exchanging equipment forproviding cooling water, and also a cooling tower 1020 portion thatprovides for circulation of air through an area where water may beflowing. In other implementations, the system 1075 and a broader coolingplant may be made up of modular units, such as a base unit that containsmany of the pumps and heat exchanging equipment for providing coolingwater, such as in a rectangular steel frame that is capable of beingshipped on a standard train bed or truck bed (e.g., in the approximatedimensions of an inter-modal shipping container). One or more separatetower sections may later be placed on top of the base unit or baseunits.

The tower 1020 may generally receive water from the system 1075 by twoalternative routes, as controlled by control valve 1025. Via a firstroute, which is used when the tower 1020 is operating in annon-evaporative mode, the cooling fluid may be routed through coilspositioned in the tower 1020 that may have air drawn through them by amechanical fan 1030, such as an updraft or downdraft fan. In such anarrangement, the fluid stays in a closed system, much like a carradiator system. Via a second route, the fluid may be expelled toatmosphere higher in the tower 1020 and made to cascade in the open overthe outside of the coils. During such cascading, the water will bebroken up so that it has more surface area exposed to the ambient air,and is more amenable to evaporation, and the great heat absorption thataccompanies evaporation. When the water finally falls to the bottom ofthe tower 1020 where it can be picked up by piping 1015, it will be muchcooler than its entering temperature.

Cooling in the non-evaporative mode can be supplemented in certaincircumstances. For example, a tank of clean water 1040 (e.g., deionizedor distilled water) may be installed near the cooling plant. Whendesired, the clean water may be supplied to sprayers 1050 located withinthe cooling plant, which can be used to assist in cooling the air toprovide for additional heat load capacity by application and evaporationof the spraying water in the air flow and on piping, etc. Such sprayingor misting may bring down the temperature of the coils through which thecooling fluid is circulating and may in turn increase the rate of heatflow out of the cooling fluid. Supplementation may also occur in eithermode, such as by the operation of a chiller or similar mechanism. Also,both supplemental options may be used in conjunction with each other,and other additional cooling options.

In general operation, the cooling plant may provide sufficient coolingfrom the cooling tower/heat exchanger/cooling coil system, though apowered refrigeration system such as a chiller may be provided for peakloads, such as when the outdoor ambient dew point is very high and thecooling tower cannot provide sufficient cooling alone. Thus, as with theair cooling mode, chiller units, water spraying, or other options mayalso be used to assist in water cooling. Control parameters for thesystem may also be set so as to avoid most or any need for the use ofchillers or other such items.

The hybrid towers provide for additional options in cooling the coolingfluid from a data center. For example, the air cooled, non-evaporative,mode may be used when the air temperature will be below a certaintemperature for a sufficient period of time, while the water cooling,evaporative mode may be used for air temperatures above a certain point.As another example, air cooled mode may be used when the requirementsfor cooling are less, and water cooled mode used for greater coolingrequirements. For example, if a portion of a data center load is downfor maintenance, or if a portion is currently being installed, or forother reasons, the cooling required by the data center may be lessenedand operating in air cooled mode may provide for sufficient cooling.

In operation, cooling plant 1010 may respond to signals from varioussensors placed in the data facility. The sensors may include, forexample, thermostats, humidistats, flowmeters, and other similarsensors. In one implementation, one or more thermostats may be providedin warm air capture plenums in the cooling modules, and one or morethermostats may be placed in workspace. The cooling plant may beoperated in response to these and other signals.

In some implementations, multiple hybrid cooling plants may be providedfor a data facility. The plants may be associated with a set number ofserver rows inside the facility, such as a single row or two rows. Theparticular plant may then serve that row or rows or servers for cooling.Additional diversity and redundancy may also be provided by connectingeach of the cooling plants to a common header and each of the rows to acommon header, so that every row can still access cooling water from theheader even if one of the plants goes off line, or if less than fullcapacity of the cooling plants is needed.

The descriptions here are exemplary only, and are not intended to belimiting in any manner. In addition, the logic flows depicted in thefigures do not necessarily require the particular order shown, orsequential order, to achieve desirable results. In addition, other stepsmay be provided, or steps may be eliminated, from the described flows,and other components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

What is claimed is:
 1. A method of cooling a data center, the methodcomprising: locating a plurality of fan coil units for cooling computersin a data center in substantially parallel rows, at least a portion ofthe rows comprising fan coil units arranged with corresponding rows ofracks that support at least a portion of the computers, with aparticular row of racks adjacent to and accessible from ahuman-occupiable workspace, and a particular row of fan coil unitsserving multiple rows of racks; positioning cooling fluid supply andreturn pipes proximate to the fan coil units in a plurality of fluidlyisolatable sections; serving a first group of fan coil units withinadjacent rows of fan coil units of the plurality of rows of fan coilunits with a first section of the plurality of fluidly isolatablesections; and serving a second group of fan coil units within theadjacent rows of fan coil units of the plurality of rows of fan coilunits with a second section of the plurality of fluidly isolatablesections different than the first section, where a particular section inthe plurality of fluidly isolatable sections provides cooling fluid formore than one row of fan coil units independently of the other sectionsin the plurality of sections, and a particular row of fan coil unitsreceives cooling fluid from more than one section in the plurality offluidly isolatable sections independently of the other rows of fan coilunits.
 2. The method of claim 1, further comprising connecting coolingfluid supply and return pipes from adjacent rows to serve a single rowof fan coil units.
 3. The method of claim 1, further comprisingconnecting the fan coil units to an overhead electrical distributionsystem.
 4. The method of claim 1, further comprising fluidly connectingthe sections to supply and return headers so as to permit sharing ofcooling fluid between the sections.
 5. The method of claim 4, furthercomprising: closing valves in one section to isolate the section fromother sections; and performing maintenance on the isolated section whileoperating substantially all of the computers in the data center.
 6. Themethod of claim 4, further comprising fluidly connecting a plurality ofcooling plants that comprise cooling towers to the supply and returnheaders, so as to provide diversity in cooling fluid supply to the datacenter.
 7. The method of claim 1, further comprising: providing coolingfluid from more than one section of the plurality of sections to a groupof fan coil units; and cooling computers supported by a particular rackin a row of the racks with the provided cooling fluid.
 8. The method ofclaim 1, wherein the fan coil units are closely coupled betweencorresponding rows of racks.
 9. A data center cooling system,comprising: a plurality of computer racks arranged in a plurality ofsubstantially parallel rows, with a particular row separated fromanother row by a human-occupiable workspace from which the computerracks in the particular row can be serviced; fan coil units arranged insubstantially parallel rows to cool air warmed by the computer racks,with a particular row of fan coil units serving multiple rows of racks;and cooling liquid supply and return pipes that are divided by isolationvalves into a plurality of cooling sub-loops, where a particularsub-loop in the plurality of cooling sub-loops provides cooling liquidfor more than one row of fan coil units, and a particular row of fancoil units receives cooling liquid from more than one sub-loop of theplurality of sub-loops, the cooling liquid supply and return pipes arearranged in a plurality of substantially parallel rows, and liquid fromone cooling liquid supply piping serves multiple rows of fan coil units.10. The data center cooling system of claim 9, where a particular row ofthe fan coil units is closely coupled between two rows of computerracks, with the human-occupiable workspace on sides of the rows ofcomputer racks that are opposed to the row of fan coil units.
 11. Thedata center cooling system of claim 9, further comprising an electricaldistribution system arranged above the plurality of racks.
 12. The datacenter cooling system of claim 9, where the cooling liquid supply andreturn pipes are arranged in a plurality of substantially parallel rowsand each fan coil unit in a row is liquidly connected to one and onlyone sub-loop.
 13. The data center cooling system of claim 12, where therows of the cooling liquid supply and return pipes are substantiallyparallel to the rows of the fan coil units, and pipes are connected sothat liquid from one row of cooling liquid supply and return pipes canserve multiple rows of fan coil units, where the rows of fan coil unitsare on opposed sides of one or more human-occupiable workspaces.
 14. Thedata center cooling system of claim 12, where the rows of the coolingliquid supply and return pipes are substantially orthogonal to the rowsof the fan coil units and human-occupiable workspace, and the pipes arearranged so that liquid from one row of cooling liquid supply and returnpipes can serve multiple rows of fan coil units.
 15. The data centercooling system of claim 9, further comprising supply and return headersthat connect the plurality of rows of cooling liquid supply and returnpipes to each other.
 16. The data center cooling system of claim 15,further comprising shut off isolation valves proximate to a connectionof each supply or return pipe to the headers.
 17. The data centercooling system of claim 15, further comprising a plurality of coolingplants that comprise one or more cooling towers and one or more chillersand are connected to the supply and return headers, so as to providediversity in cooling liquid supply to the system.
 18. A data center,comprising: a modular housing; a plurality of computer racks arranged ina plurality of substantially parallel rows in the modular housing, witha particular row separated from another row by a human-occupiableworkspace from which the computer racks in the particular row can beserviced; a cooling system that comprises: fan coil units closelycoupled to the computer racks and arranged in substantially parallelrows to cool air warmed by the computer racks, with a particular row offan coil units serving multiple rows of racks; and cooling fluid supplyand return pipes that are divided into a plurality of cooling sub-loops,where a particular sub-loop in the plurality of cooling sub-loops isarranged to provide cooling fluid for more than one row of fan coilunits independently of the other cooling sub-loops in the plurality ofsub-loops, and a particular row of fan coil units is arranged to receivecooling fluid from more than one sub-loop of the plurality of sub-loopsindependently of the other rows of fan coil units, and a particularcomputer rack in a row of the plurality of substantially parallel rowsis positioned to be cooled with cooling fluid provided from more thanone sub-loop of the plurality of cooling sub-loops, and a first sub-loopof the plurality of cooling sub-loops is arranged to serve a first setof fan coil units within adjacent rows of fan coil units of theplurality of rows of fan coil units and a second sub-loop of theplurality of cooling sub-loops arranged to serve a second set of fancoil units within the adjacent rows of fan coil units of the pluralityof rows of fan coil units.
 19. The data center of claim 18, furthercomprising: a supply header fluidly connecting the plurality of coolingfluid supply pipes; and a return header fluidly connecting the pluralityof cooling fluid return pipes.
 20. The data center of claim 18, furthercomprising a raised floor system that extends within at least a portionof the modular housing and is operable to support the fan coil units andthe computer racks, the raised floor system comprising electrical wiringraceways and cooling fluid piping pathways.